This is the first article of a (I hope) long series of articles about
'The Basics: What everyone should know about' :)
The calling convention defines the way a function or a piece of code
should arrange data before calling a function, and what to do after. It
responds to questions like "In which order should I pass the arguments
?", "Should I clean something ?", "Where is the result ?", ...
There is a lot of different calling conventions. Here are the 3 I see
the most of the time:
cdecl convention
The cdecl convention is the default one used when working with a C
compiler like GCC or MSVC. To use the cdecl scheme for a function,
you can use this syntax (GCC):
__attribute__((cdecl)) int function(int arg1, int arg2, ...);
GCC will produce the following code when calling a cdecl function
with 4 arguments :
push 0x4 ; arg4
push 0x3 ; arg3
push 0x2 ; arg2
push 0x1 ; arg1
call _cdecl_fct
add esp,0x10
mov DWORD PTR [ebp-0x4],eax
As you can see, arguments are pushed into the stack in right to left
order, and it's up to the caller to remove the arguments from the stack
(Here this is done by add esp, 0x10). The result of the function is
stored in the EAX register.
stdcall convention
The stdcall convention is the one used by Win32 APIs. It's also the
easyest to use when writing ASM code, in my opinion. A function can be
declared as a stdcall function in C with this syntax (GCC):
__attribute__((stdcall)) int function(int arg1, int arg2, ...);
GCC will produce the following code when calling a stdcall function
with 4 arguments :
push 0x4 ; arg4
push 0x3 ; arg3
push 0x2 ; arg2
push 0x1 ; arg1
call _stdcall_fct@16
mov DWORD PTR [ebp-0x4],eax
As for the cdecl calling style, arguments are pushed from right to
left, but in stdcall mode, the caller doesn't have to clean the
arguments from the stack after calling the function. A stdcall
function removes arguments from the stack before returning. This is done
by using the ret n instruction most of the time.
Like for cdecl, result is in EAX.
fastcall convention
The fastcall convention is not standardized, but we will watch the
way GCC and MSVC handle it. A function can be declared as a fastcall
function in C with this syntax (GCC):
__attribute__((fastcall)) int function(int arg1, int arg2, ...);
GCC will produce the following code when calling a stdcall function
with 4 arguments :
push 0x4 ; arg4
push 0x3 ; arg3
mov edx,0x2 ; arg2
mov ecx,0x1 ; arg1
call @fastcall_fct@16
mov DWORD PTR [ebp-0x4],eax
As you can see, not all the arguments are pushed into the stack. The
first two arguments are passed via the ECX, for the first argument, and
EDX, for the second argument. The remaining arguments are pushed into
the stack from right to left. The called function has to pop the
arguments from the stack before returning, like for stdcall.
The result is, as usual, in EAX :)
There are comments.